By Evgenia "Jenny" Nitishinskaya and Delaney Granizo-Mackenzie
Notebook released under the Creative Commons Attribution 4.0 License.
Data is sometimes more useful when presented in a summary. Although there is more information contained in the complete list of observations, a summary that makes the data more comprehensible may be more useful. For instance, we often cite the mean of a set of data instead of listing out all of the values. This is particulary useful when we have a very large number of data points.
A frequency distribution consists of:
Talking about data in intervals often makes more sense than using specific values. For instance, if all 50 observations of a random variable have landed in the interval [99, 101], we expect the next observation to be approximately 100. We do not expect it to come up with any particular value, however, and would not bet that it will be exactly equal to 100.72.
When constructing a frequency distribution, we use some number $k$ of equally sized bins. Each of the $k$ intervals has length (range of data)/$k$.
The usefulness of frequency distributions can be seen below. Binning the data shows that the observations are sampled from a normal distribution much better than the one that marks each data point individually.
In [8]:
import numpy as np
import matplotlib.pyplot as plt
# Draw points randomly from a normal distribution
X = np.random.randn(150)
print np.sort(X)
It's difficult to tell that the data is approximately normally distributed just by looking at the list above. Below, we sort them into 300 bins; this is more than the number of data points, but many are empty while some have several items because the data is clustered around the mean.
In [15]:
# Print list of frequencies of data points
# Second argument is number of bins
# Second output is the locations of bin dividers but we ignore it because there are 301 of them
hist, _ = np.histogram(X, 300)
print 'Data frequencies:', hist
Frequency distributions are often used in graphing because binned data is visually more comprehensible. For example, the commonly-used graphing library matplotlib.pyplot has a built-in histogram function which requires the number of bins $k$.
In [5]:
plt.hist(X,300);
This is still a little noisy, so we can reduce the number of bins (and increase the bin size) further to get a prettier histogram.
In [13]:
# Print frequency of data per bin
hist, bins = np.histogram(X, 15)
print 'Bin dividers:', bins
print 'Binned data frequencies:', hist
plt.hist(X,15);
However, if bin sizes are too large, the data ceases to be very informative:
In [14]:
# Print frequency of data per bin
hist, bins = np.histogram(X, 4)
print 'Bin dividers:', bins
print 'Binned data frequencies:', hist
plt.hist(X,4);